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I. INTRODUCTION 


A. BACKGROUND 

This thesis is an early step in a long term development 
project dedicated to producing the personal computer based 
Expert System Advisor for Aircraft Maintenance Scheduling 
(ESAAMS). The development of expert systems has rapidly 
expanded in the last decade. These systems are software 
progrems which solve problems requiring primarily human 
expertise. It may be both benefici.l and cost effective to 
develop and provide such an expert system to organizational 
level aviation maintenance managers to advise and assist 
them with the challenging task of aircraft maintenance 
discrepancy scheduling. This thesis builds directly upon 
the earlier work of McCaffrey (Ref. 1], and to a lesser 
extent upon the works of Chase (Ref. 2] and Allan and 


McSwain [Ref. 3]. 


B. OBJECTIVES 

The objectives of this thesis are to review the 
requirements of developing an expert system knowledge base 
and to determine the suitability, availability, accuracy and 
reliability of the Naval Aviation Logistics Data Analysis 
(NALDA) databases as the foundation of unique databases 


which are essential elements of such an expert system. 








‘4. 2 following research questions will be addressed: 


® What is an expert system, how is one developed, and how 
will they assist maintenance managers? 


@ What is NALDA, where does it obtain its data, and what 
type of information does it maintain? 


® How timely and accurate is the data contained within the 
various NALDA databases? 


® How often will the information retrieved from the NALDA 
databases need to be updated. 


® Will different databases be required to support 
organizations operating in different geographical 
locations? 

@® Can the required data be broken out of the NALDA 
databases in a format suitable for expert system 
utilization? 

C. METHODOLOGY 

Research into expert systems and various aspects of the 
NALDA system was conducted in literature, by telephone 
conversation with Mr. Bob Zolio of the Aviation Supply 
Office (ASO-045401) and members of the NALDA Users 
Assistance Group, and by a personal unstructured interview 
with Mr. Gene Woodburn (NAMO-622C). 

In addition to this analytical research, a rudimentary 
quantitative analysis was conducted on a data sample 
obtained from NALDA's Fleet Oriented Jobs (FOJ) database. 
The data sample provided various data elements concerning 
approximately 78,000 individual maintenance actions 


performed on various systems and sub-systems of Navy and 


Marine Corps F/A-18 aircrart over a twelve month period 











ending in August of 1990. Detailed information concerning 
how the data sample was developed, definitions of the 
various data elements are contained in chapter VI. A print- 
out of a small portion of the sample data is contained in 
Appendix A. Copies of the data sample diskette and printed 
results of the two analysis of variance (ANOVA) examinations 
performed will be maintained on file with the author and 
Professor Martin J. McCaffrey at the Naval Postgraduate 
School, Monterey, CA. Data manipulation and analysis was 
conducted on the Naval Postgraduate School (NPS) mainframe 
computer using the Statistical Analysis System (SAS) 
progran. 

Analysis of variance was performed on mean elapsed 
maintenance time (EMT) computed for various groups. EMT was 
chosen, based on the author's personal experience, as the 
most important predictor used in maintenance discrepancy 
scheduling. An expert system that accurately projects the 
EMT for a particular maintenance action would allow the 
maintenance manager to prioritize the discrepancies to be 
worked upon, to minimize idle time during the repair cycle, 


and to maximize utilization of his activity's limited repair 


facilities. 








D. 


THESIS ORGANIZATION 


The remaining chapters of the thesis are as follows: 


II. EXPERT SYSTEMS. A general description of expert 
systems, their components, and their development. 

Emphasis is placed upon the knowledge base and the task of 
knowledge acquisition as they pertain to the development 
of the ESAAMS project. 


III. NALDA. A general description of the NALDA systen, 
its various databases, and the source of its data. 


IV. DATABASE ACCURACY. The accuracy and consistency of 
the NALDA databases is closely examined. 


Vv. CONCLUSIONS. 





II. AN INTRODUCTION TO EXPERT SYSTEMS 


A. INTRODUCTION 

This chapter serves as an introduction to expert 
systems, with emphasis on the knowledge base component of an 
expert system and the knowledge acquisition process as they 
pertain to the proposed ESAAMS system. The concepts and 
basic elements of an expert system, how an expert system 
would benefit aircraft maintenance managers, how expert 
systems differ from conventional computer programs, the 
stages in the development of an expert system, a variety of 
knowledge acquisition techniques, and the topic of knowledge 


validation will all be discussed. 


B. EXPERT SYSTEMS 

Although the concept of expert systems has long been a 
topic within the artificial intelligence (AI) community, the 
term expert system has only recently gained wide and 
accepted use. What is an expert system? Dimitris Chorafas 


defined an expert system as: 


...software constructs that experts in specific fields 
enrich with their knowledge. By distilling their 
expertise into sets of laws and entering them into 
systems, the experts produce applications programs that 
help nonexperts solve problems in the experts' fields by 
responding to program queries. As such, expert systems 
are artificial intelligence programs that enable a 
computer to aid you in a decision-making process. The 











knowhow of the human expert is used to instruct the 
computer how to solve a problem or make a decision. 
(Ref. 4:pp. 62-63] 


Donald Waterman, a prolific writer on the subject, 


defined expert systems as: 


-..-computer programs that manipulate knowledge to solve 
problems efficiently and effectively in a narrow problem 
area. Like real human experts, these systems use 
symbolic logic and heuristics-rules of thumb-to find 
solutions. And like real experts, they make mistakes 
but have the capacity to learn from their errors. 
However, this artificial expertise has some advantages 
over human expertise: It is permanent, consistent, easy 
to transfer and document, and cheaper. In sum, by 
linking the power of computers to the richness of human 
experience, expert systems enhance the value of expert 
knowledge by making it readily and widely accessible. 
(Ref. S:p. XVII] 


Finally, M. A. Bramer defined an expert system simply 
as: 

-.-a computing system which embodies organized knowledge 
concerning some specific area of human expertise, 
sufficient to perform as a cost-effective consultant. 
(Ref. 6:p. 3] 

Bramer expanded on his definition by stating that expert 
systems usually make use of heuristic rules, relating to the 
specialty in question, which are obtained from subject 
matter experts and refined through experience. 

Heuristic is a term derived from the Greek word 
"heuriskein", meaning to discover. A heuristic is a rule of 
thumb, a technique or a simplification that limits or 
narrows the search procedure. 


The earliest acknowledged expert system, DRENDAL, was 


first developed in the 1965 [{Ref. 6:p. 3]. Two other early 











and comparatively well known expert systems, both developed 


at the Stanford Research Institute, are MYCIN (a medical 
diagnostic tool), and PROSPECTOR (a geological exploration 
tool). Another well known, and more recently developed, 
expert system is ACE (Automated Cable Expertise), developed 
by AT&T and in use since 1982. Currently expert systems are 
widely utilized in many fields and are gaining popularity in 
the military, manufacturing, banking and service industries. 
(Ref. 7:p. 69] 

Despite their growing use and the positive nature of the 
definitions above, readers should be aware that expert 
systems are not the answer to every problem and do have 
their drawbacks. Expert systems are designed to perform 
tasks that require cognitive as opposed to physical skills. 
If the task can only be learned through practicing physical 
manipulations then it is unlikely that the expert system 
approach will succeed. [Ref. 5:p. 128] The development of 
an expert system is an extremely complex and time consuming 
operation. Extracting expert knowledge is a challenging 
process. Finally, even with the best experts contributing 
to their development, knowledge based systems are not 100% 


reliable. [{Ref. 7:p. 74] 











C. DIFFERENCES BETWEEN EXPERT SYSTEMS AND CONVENTIONAL 
PROGRAMS 

Valerie Barr of the Pratt Institute writes that: 

In expert systems, unlike other types of programs, we 
tell the computer what to know, not what to do. When 
constructing the expert system, we do not provide a set 
of instructions; rather , we provide knowledge and 

advice. If we have a step-by-step algorithm for solving © 
the problem at hand, then a "traditional" program is in 
order. However, if we have no such step-by-step method, 
then an expert system is in order. (Ref. 8:p.68] 

Some readers may confuse an expert system with the more 
widely known decision support systems (DSS) or database 
management systems (DBMS). The key difference is that as 
DSSs have been constructed on the knowledge acquired through 
DBMS applications, expert systems are built upon the 
foundation of knowledge gained through the development and 
implementation of DSSs. [Ref. 4:p. 67] An easy to remember 
and rudimentary difference between expert systems and DSS or 
DBMS programs is that while the conventional programs 


manipulate data, expert systems manipulate knowledge [Ref. 


73p. 67). 


D. BENEFITS OF DEVELOPING ESAAMS 

McCaffrey determined that the aircraft maintenance 
discrepancy scheduling domain was acceptable for an expert 
system solution and that developing an expert system to 
assist aircraft maintenance managers would be beneficial to 


the U.S. Navy [{Ref. l:pp. The development of a 





107-111). 





knowledge base of shared expertise would partially solve the 
learning curve problem many managers experience when 
assigned to a squadron possessing aircraft they are not 
familiar with. 

Additionally, expert system development offers one of 
the few methods available with the potential to achieve 
significant gains in operational readiness. Because of the. 
large aircraft inventory possessed by the Navy and Marine 
Corps, a gain of as little as one percent would translate 
into an additional 50 operationally ready aircraft per day. 


{Ref. 1, pp. 110-111] 


E. COMPONENTS OF AN EXPERT SYSTEM 


An expert system consists of three basic components 


(Ref. 7:pp. 76-77]: 


@ A knowledge bank (sometimes referred to as an 
information base or a knowledge base), containing task 
specific facts and rules. Ina rule based expert 
system, the knowledge base breaks down into the rule 
base and the working memory (or data base). 


@® An inference engine, containing problem solving methods 


and a mechanism for using the knowledge base's 
information. 


® A user interface, which allows the user, the maintenance 
Manager in the case of ESAAMS, to interact with the 
progran. 
Frequently, rather than duplicate data already stored in 
existing databases within the knowledge base, an expert 


system will be designed to import data from a database when 














needed. A diagram illustrating the major components of the 
ESAAMS system, including the NALDA databases, is provided in 


Figure 1. 


MAINTENANCE 
MANAGER 


USER 
INTERFACE 


KNOWLEDGE WORKING 
BASE MEMORY 


ALDA 
DATABASES 





Figure 1 ESAAMS System Components 


The two primary components of any expert system, which 
require the majority of the development effort, are the 
inference engine and the knowledge base. The function of 
the inference engine is hypothesis proving. The inference 
engine is software that implements a search of the rules 
contained in the knowledge base, looking for matches to the 
initial information given in the working memory. The 


inference engine controls the process of invoking rules, 


10 








deciding how to apply the rules to infer new knowledge and 
in which order the rules should be applied. 

Developers of expert systems today can choose from a 
number of commercially available expert system shells. An 
expert system shell is a program containing a working, 
tested and debugged inference engine and user interface in a 
framework into which the developer can program the unique 
knowledge for his particular application into the knowledge 
base. [Ref. 8:pp. 68-69] The obvious advantage of using an 
expert system shell is that it allows the developer to 
concentrate on acquiring and refining the knowledge base 
instead of the computer code necessary to produce a 
functioning inference engine. 

The knowledge base, sometimes referred to as the 
knowledge bank in order to avoid confusion with the term 
"database", is the heart of any expert system. Walters and 
Nielsen defined a knowledge base as containing: 

---all the application-specific information. This 
information can be in the form of simple facts (e.g., 
data names and values), relationships (e.g., parent~ 
child class memberships), or procedural information 
(e.g., sequential code for printing a report or drawing 
a graph). (Ref. 9:p. 5] 

The knowledge base contains facts in its working memory, 
and rules in its rule base. Rules can be thought of as long 
term information that rarely, if ever, changes. Conversely, 


facts are short term information that can change quite 


often. Today, most commercial and experimental expert 


11 


systems use the IF-THEN rule format. Rules are formatted 
into two separate parts. The first part of the rule (IF) 
states some premise or condition. The IF part of the rule 
may contain more than a single premise and, if so, will 
contain a clause for each. These are called compound 
clauses and are linked by conjunctions AND or OR. The 
second part of the rule (THEN) states a conclusion, 
consequence, or action that will occur if the conditions of 
the first part of the rule have been met. [Ref. 7:pp. 77-78] 
For example: 

IF the animal lives in water 

AND the animal breathes water 

THEN the animal is a fish, CF 1.0 

The CF stands for certainty factor, and is sometimes 
referred to as a confidence factor or rule strength. It is 
a number between 0 and 1 representing the confidence we have 
in the validity of our conclusion. A certainty factor is 
not a probability, it is simply a number that represents the 
degree of uncertainty you have in a particular rule. 
Certainty factors are determined by the human experts 
creating the knowledge base. The certainty factors can be 
based upon either intelligent estimates, statistical data, 
or a combination of the two. In rules with compound premise 
clauses connected by either AND or OR, each clause may have 
its own certainty factor and the programmer must determine a 
means to compute a composite certainty factor for the rule. 


(Ref. 7:pp. 86-88] 
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F. EXPERT SYSTEM DEVELOPMENT 


There are several methods for developing expert systems. 
This report will summarize a scheme published by Juan J. 
Ferrada and John M. Holmes. [Ref. 10:pp. 35-41} The 
development of an expert system can be simplified to four 
global steps: 

@ Defining the problem 

@ Preliminary prototyping 

® Developing the expanded prototype 
® Delivering the system 

1. Defining the Problem 

The objectives of this step are to ensure that: the 
problem to be solved is clearly defined; the project will 
satisfy a real need; the project is technically feasible; 
the functions to be carried out are specified; the 
knowledge base requirements are established; and the 
available expert system development tools are analyzed. 

After the above questions have been answered, a decision 
must be made whether to use an expert system shell or an AI 
computer language. Expert systems can be constructed 
utilizing any number of commercially available expert system 
shells or AI computer languages. An expert shell has the 
advantage of containing a substantial amount of AI code and 
an inference engine that has been tested, debugged and 


maintained. Building an expert system using an AI computer 
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language provides the expert system developer wich greater 


flexibility while simultaneously tasking him with writing a 
considerable amount of code. 

The final requirement of this initial step is to define 
the available sources of expert knowledge. A knowledge 
engineer is the person responsible for gathering and 
organizing the expert knowledge into an expert system 
knowledge base. The knowledge engineer must identify the 
knowledge base recuirements as well as the reliability and 
availability of human experts, databases, spreadsheets, 
descriptions in text files, graphics and external computer 
programs. The topic of knowledge acquisition will be 
discussed in greater detail later in this thesis. 

2. Preliminary Prototyping 

The principal objective of this step is to quickly 
demonstrate the technical and economic feasibility of the 
expert system under development. Preliminary prototype 
testing should be analyzed to determine how well the 
interface relates to the knowledge base, how well the 
inference engine can adapt to the way the knowledge has been 
introduced to the expert system, how well the knowledge base 
has been organized by the knowledge engineer, and how well 
the proposed environment for delivery (PC, minicomputer or 
mainframe) performs. As recently as 1987 most writers 


concluded that useful expert systems had to implemented and 
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delivered on minicomputers or mainframes [Ref. 7:p. 74]. 


Currently personal computers, which are quickly migrating to 
the 386, the 486 and even more powerful chips, have begun to 
represent the single largest segment of the expert system 
business and are the key to the commercial acceptance of 
expert system technology [{Ref. 1l:p. 56]. 

3. Developing the Expanded Prototype 

The objective of this step is to develop a system 
capable of convincing management that it is a useful and 
potentially valuable tool. The expanded prototype should 
include an expansion of the knowledge base, sophistication 
of representation, and links to other systems and programs. 
The system should be capable of dealing with a variety of 
exceptions and special cases ignored during earlier 
prototyping. If the expanded prototype performs well the 
development of a more extensive delivery system may not be 
required. 

4. Delivering the System 

During this phase the developer analyzes the different 
delivery environments mentioned above and optimizes the 
program requirements in terms of memory usage or 


performance. 


G. KNOWLEDGE ACQUISITION AND REFINEMENT 
Expert systems are powerful and valuable instruments 


because of the capacity they gain from the knowledge they 
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incorporate. The development of a reliable and accurate 
knowledge base is the primary goal of any knowledge engineer 
tasked with developing an expert system. Knowledge 
acquisition and refinement have historically accounted for a 
major portion of the overall development expense and time of 
most expert svstems. Knowledge acquisition is defined as: 
...the transfer of expertise from a human expert to a 
machine. When the machine, extends its initial 
knowledge by learning methods, we refer to this as 
knowledge refinement [Ref. 12:p. 320]. 

As the demand for expert systems has multiplied, the 
limited availability of qualified and skilled knowledge 
engineers has made knowledge acqiisition appear to be a 
major bottleneck in the construction of new expert systems. 
However, this assumption is based on the dated view that 
knowledge engineering is a nana craft, depending on the 
skill of a knowledge engineer who is armed with only a pen, 
paper and tape recorder. The key to solving this knowledge 
acquisition bottleneck is to automate some portion of the 
knowledge engineering tasks. [Ref. 13:p. 49] 

Traditionally, knowledge acquisition has been performed 
by a knowledge engineer who interviews an expert, identifies 
the components of his knowledge, and builds the expert 
system knowledge base. The knowledge engineer's task is 
complicated by the fact that human experts have rarely 
analyzed their thoughts and the structure of their 


knowledge, and can rarely provide an overall account of how 
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a decision is made. Traditionally knowledge obtained from 
any source other than a human expert was discounted when 
developing an expert system. Frenzel stated, in 1987, that: 

-knowledge comes in many forms. It can be standard 

textbook knowledge that you can dig out of books, 
articles, and other references quickly and easily. This 
knowledge is important, but is usually not the best kind 
of knowledge for an expert system. The real knowledge 
will come from individuals who are experts in the 
subject. [Ref. 7:p. 105] 

The traditional interview approach to knowledge 
acquisition has several shortcomings. One is the afore 
mentioned shortage of skilled knowledge engineers. Another 
is the growth in size of expert systems knowledge bases. 
Early expert systems often used less than 100 rules, and the 
use of more than 300 rules was rare [Ref. 14:p. 44). 

Current state-of-the-art expert systems may contain tens of 
thousands of rules in their knowledge bases [Ref. 15:p. 15}. 
McCaffrey estimated the knowledge base of the proposed 
expert system would contain approximately 1000 to 2000 rules 
(Ref.1:p. 122]. Clearly automating some or all of the 
knowledge acquisition process is called for in order to 
overcome the "knowledge engineering bottleneck." 

Parsaye, who defines knowledge acquisition as "the 


transfer of problem-solving expertise from some knowledge 


source to a program," identifies three basic approaches to 
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knowledge acquisition currently available (Ref. 13:p.50}: 
® interviewing 
@ learning by interaction 
@ learning by induction 

1. Interviewing 

This technique involves the knowledge engineer acquiring 
knowledge from a human expert through a sequence of 
interviews and then encoding the acquired knowledge in the 
expert system's knowledge base. The knowledge engineer 
plays a central role in this process and his skills largely 
determine the quality of the information obtained and 
entered into the knowledge base. 

Most readers will recognize that while the interview 
process is an easy-to-apply and flexible approach, it also 
contains some serious drawbacks: interviewing is a time 
consuming and subjective process; often the knowledge 
engineer, with only a limited amount of subject matter 
expertise, will not explicitly understand important 
concepts, resulting in an incomplete or inaccurate knowledge 
base structure; and directly asking questions of experts 
risks altering their view of what they actually do. With 
few, if any, knowledge engineers familiar with the complex 
task of aircraft maintenance scheduling, these drawbacks 
will be particularly applicable to the development of the 


ESAAMS project. 
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2. Learning by Interaction 

This knowledge acquisition technique relies on computer 
assistance. In essence the human expert interacts directly 
with a computer program that captures his expert knowledge. 
This process focuses on specific interactive methodologies 
and interviewing techniques that help experts clarify the 
structure of their own thoughts and knowledge. The need for 
a knowledge engineer can be significantly reduced by 
utilizing the learning by interaction technique. 

3. Learning by Induction 

In this knowledge acquisition process a computer program 
distills knowledge by examining data and examples. The 
basic concept is to have the program learn to perform a task 
by analyzing the data documenting the human expert's 
performance. The primary difficulty with learning by 
induction is identifying suitable attributes and 
characteristics on which to perform the induction. The 
primary advantages of this technique are that dependence on 
knowledge engineers and human experts is diminished, and 
knowledge bases containing large numbers of rules can be 
developed in a short period of time. 

4. Summary 

All three knowledge acquisition techniques have their 


advantages and disadvantages. The majority of expert 


systems currently under development will use all three 











techniques in the development of their knowledge bases. The 
one factor all three techniques have in common, and an 
important source of expert knowledge, is historical data 
about how an expert actually performed a task. Much of this 
data is contained in computerized databases, such a the 
various NALDA databases. A fundamental difficulty with 
using any of these databases is the obstacle of connecting 
the AI computer language or the expert shell to various 
external programs and databases. William Martorelli expands 
on this challenge: 

For expert systems to be most useful, they should be 
able to communicate with existing systems and data 
sources. Today's PC-based (expert system) applications 
are still often stand-alone entities, or they possess 
rudimentary links to spreadsheets or PC databases. [Ref. 
llsp. 62] 

A number of vendors are striving to make their expert 
system development tools more readily connectable to 
external programs and data bases. Ken Pedersen synopsized 
the connection capabilities of 11 commercially available Pc- 
based expert system shells in Reference 16. Readers who 
desire a more thorough documentation of the connection 
capabilities of these shells are referred to The PC Expert 
Systems Shoot-Out, published by Expertise Associates of W. 
Lafayette, Indiana. In general, many of the shells 


available on the market today can connect to such popular PC 


based programs as Lotus 1-2-3 and dBase. Many of the expert 


system shells, such as VP-Expert, can offer an induction 








module capable of translating record based examples into 


production rules. The resulting knowledge base is only as 


good as the data file on which it is based. 


H. KNOWLEDGE VALIDATiON 
A fundamental question concerning every expert system is 
how applicable is the expert knowledge acquired for your 
system? Knowledge base validation is an area of great 
concern to all knowledge engineers. Validation of the 
knowledge base should be viewed as an ongoing process built 
into the development effort. The two primary goals of any 
software validation effort are to ensure that: (1) the 
program performs the functions intended, and (2) the program 
does hot perform any intended function that could adversely 
affect the performance of the entire system. (Ref. 17:p. 
287] Parsaye expands on this topic: 
Intuitively the aim of any validation effort is to 
ensure that the expert system behaves correctly. but 
what does this mean for systems that produce inexact 
results? Can we ever be 100% sure the knowledge is 
correct? 
From a philosophical viewpoint the answer is no. Even 
in traditional programs, you can never be 100% sure 
about a verification effort in all cases since as the 
program size grows the size of the verification program 
will also grow, and at some point the verification 
system may be more complex and error prone than the 
program itself. [Ref. 13:p. 59] 
Parsaye further states that the two key questions to ask 
during validation are: (1) How often are mistakes made?, 


and (2) What is the price to be paid for each mistake? 
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Parsaye's approach is to compare the performance of the 
expert system under development against that of a human 
expert, or even another expert system, on a series of test 
cases. Because the overall measure of performance obtained 
under these circumstances would depend largely on the 
quality of the test cases, it is important to also measure 
the quality of the test cases. Utilizing a more 
representative set of test cases will furnish the developer 


with a more significant set of results. 


I. SUMMARY 

This chapter has provided the reader with a large amount 
of information concerning expert systems and their 
development. The three most significant points to take from 


this chapter are that: 


® With the ready availability of commercially proven 
shells, the most challenging task facing today's expert 
system developer is to acquire, develop and validate the 
applicable knowledge base rules. 


@® Whichever combination of knowledge acquisition 
techniques is used, a data base of information 
concerning past expert performance is required. The 
quality of the knowledge base developed will depend 
largely on the accuracy, timeliness and completeness of 
the information contained in the database. 


® Connectability with the NALDA system should be 
considered when choosing an expert system shell for use 
on the ESAAMS project. Because NALDA has the capability 
to download data via floppy diskette in ASCII format, 
indirect connection through widely available software, 
such as dBase or Lotus 1-2-3, with many currently 
available expert system shells is likely. Further 
research into the direct connection of the NALDA system 
with an expert system shell is warranted. 
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III. NALDA 


A. INTRODUCTION 

This chapter describes the NALDA system, including the 
source of NALDA's data and the various databases it 
maintains. The following specific questions will be 
addressed: What is NALDA? Where does NALDA obtain its 
data? What type of information is stored in the various 
NALDA databases? Which database is the most likely source 
of information for the unique databases which will become 


essential elements of the ESAAMS system. 


B. NALDA DEFINED 
The NALDA system, is: 
-.--an automated database and information retrieval 
system for aviation logistics management and technical 
decision support. Analysis capability is provided 
tnrough interactive query and batch processing from 
remote terminals. [Ref. 18:p. I-1] 
The principal objective of the NALDA system is to supply 
a state-of-the-art management information system to assist 
Naval Aviation maintenance and logistics managers in making 
improved decisions affecting U.S. Naval Aviation readiness. 
To accomplish this objective the NALDA system provides a 
centralized data bank, including remote terminals with 


maintenance data retrieval and analysis capabilities, that 


can solve complex integrated logistics support problems for 
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which other available means have consistently proved 
inadequate. [Ref. 18:p. I-1] Overall management of the 
NALDA program is conducted by the Naval Air Systems Command, 
while day-to-day operations are coordinated by the Naval 
Aviation Maintenance Office (NAMO), located aboard Naval Air 


Station Patuxent River, Maryland. 


C. NALDA DATA SOURCES 

NALDA incorporates data from an assortment of different 
sources including Naval Aviation Depots, the Aviation Supply 
Office, and the Naval Safety Center [Ref. 18:p. I-1]. The 
principal source for NALDA data is the monthly Aviation 
Maintenance and Material Management (3-M) summary produced 
by the Naval Aviation Maintenance Support Office (NAMSO). 

The source for the 3-M summary is the Navy's Maintenance 
Data System (MDS). The Naval Aviation Maintenance Plan 
(NAMP), separates the MDS data flow into three cycles (Ref. 
19:para. 2.1.3 - 2.1.7]: 

@ Local Cycle: During this cycle data elements concerning 
maintenance actions performed are entered onto source 
documents by technicians at either the organizational or 
intermediate level. 

@ Local/central Cycle: During this cycle the completed 
source documents are screened, corrected, and converted 
into machine language by a supporting data services 
facility (DSF). 

® Central/external Cycle: During this cycle data from the 
numerous DSFs are collected and processed by NAMSO, and 


various reports are issued to originating activities and 
the chain of command. 
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1. Local Cycle 

The local cycle of the MDS is designed so that each 
technician, when performing a task, converts a narrative 
summary of the task into a series of alphanumeric and 
numeric codes and enters the coded information on to one of 
two source documents, either a Support Action Form (SAF) or 
a Visual Information Display System/Maintenance Action Form 
(VIDS/MAF). Source documents are screened for accuracy and 
completeness by the technician's work center supervisor, 
maintenance control supervisor, and the activity's data 
analyst. These source documents are then collected and 
transported to a local data services facility (DSF), located 
aboard either the host air station or deployed ship, where 
the information is converted to machine record. The DSFs 
screen the incoming source documents and any questionable 
data are referred back to the originating activity for 
verification. The DSF uses the machine records to produce a 
number of periodic reports summarizing the submitted data. 
These reports are used to verify the data, for local unit 
analysis, and for maintenance planning. 

2. Local/Central Cycle 

After verification, duplicate record files of the 
information are generated and transmitted to NAMSO, Naval 
Aviation's central collection facility. At NAMSO the data 


received is combined with data received from other DSFs. 
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Machine runs for errors which are detectable by computer are 


made, and the results of this error analysis are forwarded 
back to the DSFs and originating activities. 

3. Central/External Cycle 

In addition to providing periodic 3-M feedback reports 
to the originating activities, NAMSO provides data and 
reports to various agencies, including the Chief of Naval 
Operations (CNO), various systems command field agencies, 


NALDA, and contractors. 


D. NALDA DATABASES 

NALDA maintains a number of specialized processing 
systems and databases. These systems and databases are 
designed to provide system users with the necessary 
information to solve problems and to make informed 
management decisions. [Ref. 18:pp. I-2 to I-3] These 


databases include: 


@ Fleet Oriented Jobs (FOJ): contains the most recent 
eighteen months of selected maintenance, material, 
readiness, inventory and operations data and are the 
principal repository of source data in the NALDA systen. 


® Equipment Condition Analysis (ECA): Provides users with 
3-M data and flight data, as far back as 1974. 


® Technical Directive Status Accounting (TDSA): Stores, 
maintains, and disseminates information concerning the 
incorporation/non-incorporation of Technical Directives. 
Provides projected and actual man-hour reporting, 
configuration status of equipment items, and change-kits 
material accounting. 


® Support Action Form (SAF): Contains the most recent 


eighteen months of support action data. 
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Depot oriented Jobs (DOJ): Contains data collected by 
the Depot Maintenance Data System related to jobs 
originated by the depots. 


Intermediate Maintenance Activity Analysis (IMA): 
Contains summary data that facilitates analysis of 
production data at the intermediate level of maintenance 
and at individual IMAs for items identified by either a 
National Stock Number or a part number. 


Utilization (UTIL): Contains the most recent eighteen 
months of utilization and aircraft flight/flight hour 
data. 


NALDA/Aircraft Engine Management System (NALDA/AEMS) : 
Contains information derived from the Navy's on-line 
AEMS. 


Equipment Summary Reports (ESR): Contains current 
summary information which is identical to the 
information contained in Aviation 3-M reports. 


Scheduled Removal Component (NEWSRC): Contains data 
from Scheduled Removal Component (SRC), Assembly Service 
Record, Module Service Record and History Record forms 
which are collected at the SRC Central Repository. 

THE MOST LIKELY SOURCE OF DATA FOR ESAAMS 


After interviewing numerous personnel working in the 


NALDA User Assistance Group, this author determined that the 


most probable source for the majority of the data needed for 


the ESAAMS system knowledge base was the FOJ database. 


Based on this determination, further research and analysis 


concentrated on the FOJ database. Other specialized 


databases maintained by NALDA could be utilized, depending 


upon the information required and the scope of the proposed 


ESAAMS systen. 


27 











FP. SUMMARY 

This chapter has presented the reader with a large 
amount of information concerning the NALDA system and the 
databases it maintains. The key point to take from this 
chapter is that the NALDA databases are Naval Aviation's 
central repository of logistical and maintenance 
information. Because they gather historical information 
from a wide variety of sources throughout the fleet, and 
because they are uniquely organized to handle ad hoc 
queries, they are the most likely source of data for the 


unique databases which will become essential elements of the 


ESAAMS system. 








IV. SUITABILITY AND ACCURACY 


A. INTRODUCTION 

This chapter evaluates the suitability and accuracy of 
the NALDA databases as the foundation for the knowledge base 
of the ESAAMS system. A description of the data sample used 
in this evaluation process is provided. The following 
specific research questions are answered: Can the data be 
broken out in a useful form? How dated is the NALDA data? 
How accurate is the data? How does geographical location of 
the originating activities affect the data? How often would 


a knowledge base founded upon the NALDA databases need to be 


updated? 


B. DATA SAMPLE 

A data sample, comprising records of approximately 
78,000 individual F/A-18 maintenance actions, was obtained 
from the NALDA Users Assistance Group. A print-out of a 
portion of this data sample is attached as Appendix A. The 
following information was contained in each record: Job 
Control Number (JCN), Work Unit Code (WUC), Transaction Code 
(TRANS), Malfunction Code (MAL), and Elapsed Maintenance 
Time (EMT). Another data element, Maintenance Man-Hours 


(MMH), was provided in the data sample but was not used in 
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the analysis. A brief description of each data element 
follows. 
1. Job Control Number (JCN) 

The JCN is a 9-, 10-, or 11-character alphanumeric code 
that serves as a base for MDR and Maintenance control 
procedures. The JCN allows for separate identification of 
each maintenance action, and provides a link with the 
maintenance actions performed by the IMA (Intermediate 
Maintenance Activity) in support of an activity or an 0- 
level (Organizational level) maintenance discrepancy. [Ref. 
19:para. 2.1.6.7] The JCN is composed of four parts: 


® Organization Code: This is a three-character 
alphanumeric code that identifies an organization. 


® Day: This is the three-character part of the Julian 
date specifying the day of the year. This is the date 
the JCN was assigned to a maintenance action and does 
not necessarily reflect the date on which work was 
actually started. 


® Serial Number: The serial number is either a three 
character number that runs sequentially from 001 to 999, 
or a three character alpha/numeric number. This number 
is normally assigned in sequences as new jobs are 
initiated. 


@ Suffix: The JCN suffix is a structured alpha/numeric 
code added to the basic JCN to identify a sub-assembly 
or sub-assembly repair action performed independently of 
the major component repair. The Suffix is used only for 
I-level (Intermediate level) maintenance functions 
regardless of where the maintenance is being performed. 


2. Work Unit Code (WUC) 


The WUC is a one, three, five or seven character numeric 


or alpha/numeric code. It identifies a system, subsysten, 





set, major component, repairable subassembly, or part of an 
end item being worked on. The system code is the first two 
positions of the WUC, and is used to identify the system 
within the aircraft/equipment on which work is being 
performed. These codes are listed in the WUC manual 
applicable to the type, model, and series of aircraft being 
maintained. [Ref 19:para. 2.1.6.12} 

3. Transaction Code (TRANS) 

This is a two-character numeric code used to identify 
the type of data being reported [Ref 19:para. 9.2.4.8]. 
Examples of commonly used transaction codes include: 

® Transaction Code 12: On-Equipment work, including 


engines, involving non-repairable components/items 
documented as failed parts. 


@ Transaction Code 11: On-Equipment work not involving 
removal of defective or suspected defective 
components/ items. 


@ Transaction Code 23: Removal and replacement of a 
defective or suspected defective repairable component 
from an end iten. 
Appendix P of reference 19 contains a complete list of these 
codes with definitions. 

4. Malfunction Description Code (MAL) 

This is a three character numeric code used to describe 
the malfunction occurring within the item, assembly or sub- 
assembly (Ref. 19:para. 9.2.4.12]. Examples of common 


malfunction codes include: 


® 782: Defective or damaged tire sidewall, tread, bead, 
etc. 
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@® 381: Leaking - internal or external. 


@ 800: No defect - component removed/reinstalled to 
facilitate other maintenance. 


@® 814: Cannibalization - lack of replacement material. 
A complete list of malfunction codes is contained in 
Appendix I of reference 19. {Ref. 19:para. 6.2.4.6.5] 

5. Elapsed Maintenance Time (EMT) 

EMT is defined as the number of clock hours involved in 
making the repair (in hours and tenths). Although EMT is 
directly related to job man-hours, it should not be confused 
with the total man-hours required to complete a job, for 
example, if four persons worked together for 2.5 hours to 
make a repair, the total maintenance man-hours expended 
would be 10.0 and the EMT would be 2.5 hours. [Ref.19:para. 
9.2.4.15] 

6. Data Sample Acquisition 

The data sample provided by NALDA was comprised of the 
15 top WUCs for the F/A-18 as determined by numbers of 
maintenance actions processed in a one year period. All 
inspection WUCs were mistakenly excluded from the data 
sample by members of the NALDA staff. This misunderstanding 
was unknown to the author until the project was near 
completion. The statistical effect of limiting the data 
sample to repair actions only, instead of the requested top 
15 WUC's, is estimated to be minor. Data elements on every 


organizational level F/A-18 maintenance action containing 
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one of these 15 WUCs, performed during the twelve month 
period ending in August of 1990, were included. 

The data sample was downloaded from the NALDA systen, 
compressed, and loaded onto a standard 1.2 MB, 5 1/4" floppy 
disk in ASCII format. Upon receipt, the data was un- 
compressed using the PKUNZIP program, provided on the 
diskette, and loaded onto the NPS mainframe computer. 
Readers interested in obtaining access to the NALDA system 
are advised to contact the NALDA Users Assistance Group at 
Autovon 356-4454. 

7. Reasoning 

Based Upon the author's personal experience and informal 
interviews with numerous other Aerospace Maintenance Duty 
Officers, the data elements listed above were selected for 
two primary reasons: 


@ The most critical piece of information required for 
maintenance scheduling is the projected EMT. 


® For comparison purposes, maintenance actions were deemed 
to be identical if they possessed the same WUC, MAL, and 

TRANS codes. 

EMT was selected for analysis because projected EMT is a 
vital piece of information that is required for efficient 
maintenance scheduling. In order to maximize their 
activity's aircraft operational readiness, organizational 
level maintenance managers seek to effectively manage their 


limited manpower and material resources. The challenge 


these managers face several times a day is to wisely 
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schedule repair actions so as to maximize the number of 
aircraft available to meet the daily flight schedule. 
Projected EMT is also vital to the effective coordination 
and management of shared resources, such as hangar bay 
spots, support equipment and special tools. 

Meeting the daily flight schedule requirements with 
safe, flyable and properly configured aircraft is the number 
one priority for all organizational level maintenance 
managers. When an aircraft returns from a mission with one 
or more discrepancies, the maintenance manager must quickly 
ascertain the nature of the discrepancies ("up" or "down")? 
and whether or not his activity possesses the capability to 
repair the aircraft. Assuming that the aircraft can be 
repaired by the sctiviey: the. maintenance manager must then 
prioritize the discrepancies and determine when to initiate 
the repair actions. Aircraft with "down" discrepancies that 
can be quickly repaired (i.e. a short projected EMT) and 
returned to an “up" status are generally the number one work 
priority, so as to provide the maximum number of aircraft 
assets to meet the daily flight schedule. "Up" discrep- 
ancies and "down" aircraft with long projected EMT will 
generally receive a lower work priority, often being 


scheduled for repair after the flight schedule has been 


1uUp discrepancies are minor problems which do not prevent the 
aircraft from safely performing its mission. Down discrepancies 
prevent an aircraft from flying until repaired. 
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completed. A more detailed description of the maintenance 


control decision environment is contained in Reference 1. 
Several rudimentary statistical analyses were performed 
upon this data sample. The author recognizes that the 
sample may not be representative of all of the information 
contained in the NALDA databases because it: (1) is limited 
to one aircraft type, (2) is composed of only the most 
common maintenance actions, and (3) analysis was further 
limited to those maintenance actions averaging at least 10 
occurrences per month. Nevertheless, the analysis did 
reveal much about the accuracy and consistency of NALDA's 


FOJ database. 


C. NALDA DATA STRUCTURE/FORMAT 

As a large relational database, the NALDA system can 
easily retrieve any required information contained in any 
one of its specific databases. While the computational 
capabilities of the NALDA system are limited (means and 
standard deviations can be calculated), the desired data can 
be easily downloaded in a variety of formats for use on 
other more powerful computers. 

The data sample used in preparing this thesis was 
assembled and downloaded onto a standard floppy disk in 
ASCII format. The NALDA User Assistance Group indicated to 
the author their willingness to provide data samples to 


support future research. Tasking NAMO to regularly organize 
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and download the NALDA data, by aircraft type, required to 


develop and maintain the unique databases which will become 
essential elements of the ESAAMS system will, in the 
author's opinion, cause only a minor addition to their 
present workload. 

Based upon this author's personal experience every 
expert likely to be interviewed during the knowledge 
acquisition phase, particularly senior enlisted maintenance 
managers, should be thoroughly familiar with the data 
elements contained in the various NALDA databases. Framing 
the expert's reasoning in terms of data elements contained 
in the various NALDA databases should prove to be a time 
consuming, but not particularly difficult, task. Some items 
that are significant factors in organizational level 
aviation maintenance scheduling, such as the availability of 
qualified technicians and the availability of required 
support equipment or hangar space, are not contained in any 


of the NALDA databases. 


D. NALDA DATABASE MAINTENANCE 

Current NALDA instructions call for the various 
databases to be updated at least monthly and set a maximum 
data time lag goal of 60 days. [Ref. 20] During interviews 
with various members of the NALDA Users Assistance Group it 
was stated that the time it takes the data concerning a 


particular maintenance action to flow through the three MDS 
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cycles, through NAMSO, and eventually in to the various 


NALDA databases rarely exceeds 90 days and currently 
averages approximately 60 days. [Ref. 21] No documentation 
was available to support this statement, however these 
figures of 60 and 90 days corroborate the earlier findings 
of McCutcheon [Ref. 22:p. 33]. 

It is important to note that the majority of the 
maintenance managers interviewed by McCutcheon felt that 
reports generated from data that was 60-90 days old were of 
little use to them in their management functions. What most 
maintenance managers desired was a real-time management 
information system. [Ref. 22:p. 33] This author knows of 
no such real-time, or even near real-time, system which will 
be in place at the organizat ione1 level in the foreseeable 


future. 


E. NALDA DATA ACCURACY 

With the understanding that the accuracy of a database 
is directly affected by the quality of data input, this 
question must be examined from two aspects: 


® how accurately is the information transferred from the 
source documents and entered into the NALDA databases? 


@ how accurately does the information retrieved from the 


NALDA databases reflect the maintenance actions actually 
performed in the fleet? 
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1. How Accurately is the Information Transferred from 
the Source Documents and Entered Into the NALDA 
Databases? 

Members of the NALDA Users Assistance Group were 
confident, during interviews with the author, that the vast 
majority of keypunch errors were caught and corrected in the 
local cycle of the MDS system. All members interviewed 
insisted that the keypunch error rate detected at NAMSO 
during the local/central cycle averaged less than 1%, 
although this figure could not be documented. [Ref. 20] The 
author, based on personal experience, estimates that the 
claimed 1% keypunch error rate is reasonably accurate. 

2. How Accurately does the Information Retrieved from 

the Various NALDA Databases Reflect the Maintenance 
Actions Actually Performed by the Fleet Activities? 

While investigating this question the author contacted a 
representative of the Aviation Supply Office (ASO) who deals 
regularly with the NALDA databases. This representative 
voiced significant dissatisfaction with the accuracy of the 
NALDA data. ASO's dieuatierscticn is based upon three 
general points (Ref. 23]: 

® Reports generated by the MDS system are used to appraise 
unit performance. Therefore it is in the originating 


units interest to "fudge" the data, particularly data 
elements affecting unit readiness. 








@® Technicians and maintenance managers see little benefit 
from the reports generated by the MDS system. This 
leads to inaccurate, inconsistent and incorrect 
reporting of maintenance activities. 


@ Many of the data elements within the MDS system are 
loosely defined and interpreted, leading individual 
activities document identical maintenance actions 
differently. 

These three points were echoed in separate interviews 
with two representatives of the Naval Air Systems Command. 
(Ref. 24 and Ref. 25] 

Although no data is available to document the first 
point, the author, based upon personal experience, will 
concede that "data fudging" does occur. The majority of 
"data fudging" occurs with Subsystem Capability Impact 
Reporting (SCIR) data elements, which are used to generate 
readiness statistics. The extent to which SCIR data will be 
utilized in the knowledge acquisition process for ESAAMS is 
unknown at this time. SCIR data was not analyzed as part of 
this thesis. 

To validate the other two points a large number of 
interview summaries, conducted by several representatives of 
the Navy Fleet Material Support Office (FMSO), concerning 
the MDS system and NALDA in general, and NALDA's TDSA 
database in particular, were obtained. This series of 
interviews was conducted with all levels of the chain of 
command, ranging from the Systems Command level down to 


representatives of an air station supply department. 


Typical of the comments made in many of the interviews were 
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those voiced by a representative of the Commander, Naval Air 
Forces Atlantic Fleet who viewed the NALDA data as 
inherently inaccurate, and cited three primary reasons for 
this inaccuracy [Ref. 26] : 
er entries are s ctured: Because part 
numbers are assigned by the manufacturer, varying widely 
in length and combination of alpha and numeric 
characters, no easy method exists to screen source 
documents or data records to detect incorrect entries or 
keypunch errors. 

® Code changes: Because many data element codes, 
particularly Work Unit Codes, are often updated and 
changed, many data entries are incorrectly made because 
the technician has memorized the old code and fails to 
check the manual. Additionally, changing data element 
codes vastly complicate the task of analyzing and 
comparing historical data. 

@ Loose definitions: Definitions of many codes, 
particularly malfunction codes, are loose. Individual 
interpretations of which data code is appropriate to a 
particular maintenance action will have a significant 
impact on the consistency and accuracy of the databases. 
Another problem concerning the accuracy of the NALDA 

databases was identified by a representative of the Naval 
Aviation Depot (NADEP) at Norfolk, Virginia. His point was 
that documentation of repair actions of individual 
components was inconsistent at most NADEPs, including his 
own, and virtually non-existent if the item was repaired by 
the manufacturer or an independent contractor. [Ref. 27] 
Because this project is not likely to be concerned with the 
repair history of individual components, and is 
concentrating on the organizational level, this problem 


should have little impact on the proposed project. 
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During another interview with a representative of the 


NADEP Norfolk A-6 engineering staff, the results of an audit 
of NALDA's TDSA database with respect to a specific 
airframes modification (AFC-562) revealed that of 1061 
entries, 130 (12%) were incorrect [Ref. 28]. Discrepancies 
fell into two general categories: 

@® TDSA showed that the change was not incorporated, but 
the change actually had been incorporated on the 
aircraft. (60 discrepancies) 

@ TDSA showed that the change was incorporated, but the 
change actually had not been incorporated on the 
aircraft. (70 discrepancies) 

It must be stressed that the deficiencies identified 
above are largely opinions based on the personal experiences 
of the interview subjects, and are not conclusive proof that 
the data contained in the NALDA system is inaccurate. It 
should also be noted that any deficiencies that may exist in 
the NALDA databases appear to be largely the result of "user 
error" throughout the fleet and mismanagement of the MDS 
program, and are not the fault or specific responsibility of 
the NALDA managers. Further evidence concerning the 
accuracy of the NALDA databases will be discussed in later 


sections of this chapter. 


F. THE AFFECT OF GEOGRAPHICAL LOCATION OF ORIGINATING 
ACTIVITIES ON EMT 
To ascertain the effect of the geographic location of 


the originating activities on EMT, the data records from 
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the sample provided by NAMO were separated, using the 
organization codes located within the job control number 
(JCN), into East coast and West cost maintenance actions. 
The mean EMT for 92 individual maintenance actions for both 
coasts were compared and the variances analyzed. The 


following definitions were established: 
Fig? Wease=Hwest 


A, : Peast* bwest 


H, is the hypothesis we wished to test and is referred to as 
the null hypothesis. The rejection of H, leads to 
acceptance of the alternative hypothesis, denoted H,. An 
example listing of the ANOVA results is contained in Figure 
2.4 

Of the 92 maintenance actions analyzed, 42 (47%) 
indicated a probability of less than 5% that the two means 
were equal. Assuming that this is a representative sample, 
the results indicate that the geographical location of the 
originating activity does significantly affect the NALDA 
databases. If we are to use historical EMT data to project 


EMT, the expert system will need to take this fact into 


complete listings of these ANOVA results will be maintained 
on file by the author and Professor Martin J. McCaffrey of the 
Naval Postgraduate School, Monterey California. 
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Mean_ EMT 
1.1311 
0.9506 





Figure 2 Analysis of variance for: WUC=13C1700, TRANS=23, 
MAL=782. 


consideration. NALDA EMT data will most likely be divided 
into East and West coast databases. 

A closer examination of the complete ANOVA results 
listings and the example contained in Figure 2 reveals two 
other interesting factors. First, despite a low statistical 
probability that the true East and West coast means were 
equal, many of the sample means differed by 0.25 hours or 
less. While a variation of 0.25 hours may translate into 
significant statistical difference between means, from a 
practical maintenance manager's viewpoint this difference is 
essentially insignificant for many maintenance actions. 

The second factor noted was the wide variation in the 
frequency of occurrence of identical maintenance actions 
between the East and West coasts. With both coasts 


containing roughly the same number of F/A-18 aircraft, the 
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number of identical maintenance actions performed on each 


coast was expected to be approximately equal. While the 
majority of maintenance actions analyzed displayed a rough 
parity in terms of number of maintenance actions performed 
on both coasts, ratios of two-to-one and three-to-one were 
common, with the highest ratio noted exceeding twenty-to- 
one. The example results displayed in Figure 1 show a near 
three-to-one ratio between the west and east coasts. 
Neither the East or West coast consistently dominated its 
counterpart when the frequencies were imbalanced. This 
inconsistency lends support to the allegations listed above 
that the definitions of some MDS code elements are loose. 
If activities performing identical maintenance actions 
continue to document them differently because of loose code 
definitions, inconsistencies and inaccuracies in the 


database will persist. 


G. UPDATE FREQUENCY OF AN EXPERT SYSTEM DATABASE, FOUNDED 

UPON THE NALDA DATABASES 

To evaluate how often the knowledge base of the ESAAMS 
system will require updates from the NALDA databases, the 
maintenance actions documented in the data sample were 
grouped by month, using the Julian date component of the 
JCN. Mean EMTs for each month for each specific maintenance 
action were compared and the variances analyzed. Months 


were numbered sequentially, one through twelve, and the mean 
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EMT for each month was calculated. The null hypothesis, 4), 


was defined as follows: 


Ho? By =Ba=- s+ = Hye 


H,, the aiternative hypothesis, was defined as at least two 
of the means not being equal. An example listing of the 


ANOVA results is contained in Figure 2.° Of 105 separate 





Figure 3 Analysis of Variance for: WUC= 62X2100, TRANS=18, 
MAL=814. 


maintenance actions examined , 32 (30%) indicated a 
probability of less than 5% that the true means of all 


twelve months were equal. The ANOVA results indicate that 


3Complete listings of these ANOVA results will be maintained 
on file by the author and Professor Martin J. McCaffrey of the 
Naval Postgraduate School, Monterey California. 
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updates of the knowledge base for at least these 32 
maintenance actions will need to be conducted more 
frequently than once per year. The actual determination of 
how often to update the databases used to develop and 
maintain the knowledge base for the proposed expert system 
will require a more detailed statistical and cost/benefit 
analysis and is beyond the scope of this report. 

It was noted that while small variations (less than .25 
hours) between the monthly mean EMTs and the mean EMT for 
the twelve month period may have been statistically 
significant for several of the maintenance actions examined, 
they were again, from a practical user's viewpoint, 


essentially insignificant for many maintenance actions. 


H. SUMMARY 
This chapter has presented the reader with a large 
amount of information concerning the accuracy and 
consistency of the various NALDA databases and their ability 
to support the unique databases which will become essential 
elements of the ESAAMS system. The key points to take from 
this chapter are: 
® Because every expert in Naval Aviation maintenance 
management is thoroughly familiar with the MDS system, a 
database, such as those maintained by NALDA, can easily 
break-out the data required to validate the expert's 
reasoning. 
@ The data contained in the various NALDA databases is 


updated at least monthly, with an average time lag of 60 
days between completion of a particular maintenance 
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action and inclusion of the data in the NALDA systen. 
While real-time, or near real-time, database updates may 
be desirable, such a system may not be cost effective 
and is not foreseen to appear at the organizational 
level in the near future. 


While the accuracy of the transfer of data from source 
documents, through the three MDS cycles, and into the 
NALDA databases appears to be very good, the ability of 
the MDS data elements to accurately reflect the actual 
maintenance actions being performed in the fleet is in 
doubt. This doubt is primarily caused by the loose 
definitions of some of the data elements, leading to 
inconsistent documentation of identical maintenance 
actions. 


Geographical location of originating activities, even 
when generalized to just the east and west coasts, has a 
substantial affect upon the data elements most likely to 
affect aircraft maintenance scheduling. 


While a large number of maintenance actions sampled 
displayed adequate consistency of mean EMT figures 
throughout a twelve month period, 30% did not. At least 
this 30%, and perhaps more, will require database 
updates more often than once per year. 
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IV. CONCLUSIONS 


A. SUMMARY 

In chapter II we learned that the task of acquiring the 
knowledge required to construct a useful and practical 
expert system is the most difficult step in the entire 
expert system development process. While a number of 
knowledge acquisition techniques are available, the growing 
size of knowledge bases is necessitating a move to automate 
some portion of the knowledge acquisition process. 

A number of expert system shells currently available on 
the commercial market offer automated knowledge acquisition 
modules. These modules possess the capability to 
automatically generate knowledge base rules based upon the 
data contained in various database files. The key to any 
automated knowledge acquisition effort is the accuracy of 
the data contained in the databases and the connectability 
of the expert system shell used to develop the system. 

In Chapters III and IV we examined the NALDA databases 
in an effort to determine their suitability for use as the 
foundation for the knowledge base of our proposed expert 
system. Several weaknesses in the source of NALDA's data, 
Naval Aviation's MDS system, were identified. Mean EMT was 


assumed to be the most critical data element for aircraft 
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maintenance discrepancy scheduling maintained by NALDA. 
Rudimentary statistical analysis revealed that mean EMT: (1) 
is significantly affected by geographical location, and (2) 
for a notable percentage of maintenance actions, varies 
significantly during a twelve month period. 

Despite the drawbacks identified above, the NALDA 
databases are uniquely qualified to serve as the foundation 
for the knowledge base of our proposed expert system. While 
the inconsistencies identified among the data may inhibit or 
preclude the automation of the knowledge acguisition 
process, it does not make the databases unusable. Many of 
the inconsistencies, while statistically significant, are 
insignificant from a practical users point of view. 

NALDA is uniquely qualified to provide the information 
required to serve as the foundation for the EMT database of 
our proposed expert system for the following reasons: 

@® As Naval Aviation's central repository of logistical and 
maintenance data, NALDA is the only conceivable source 
for much of the data required. 

@® Every aircraft maintenance expert likely to be 
interviewed during the knowledge acquisition process 
will be thoroughly familiar with the data elements 
contained in the various NALDA databases. These data 


elements can thus serve as a "common language" when 
expert reasoning are consolidated. 


@ The source of much of NALDA's data, the three MDS 
cycles, are in place and functioning throughout the U.S. 
Navy. Despite any shortcomings the system may possess, 
replacing it or duplicating it would be prohibitively 
expensive. 








@ The NALDA system is organized to respond to ad hoc data 
inquiries. Any data required during knowledge 
acquisition can be quickly retrieved from one or more of 
the various databases, and downloaded in a variety of 
communication formats. 


B. RECOMMENDATIONS 

Based upon the previous discussion, it is submitted that 
utilizing NALDA as the foundation of a knowledge base of an 
expert system to be used for aircraft maintenance 
discrepancy scheduling is feasible. The improved management 
effectiveness, and potential for improved aircraft 
operational availability, that such an expert system offers 
warrant continued research and development efforts. 

Areas that warrant further research include: 

@ Easy access to NALDA terminals would allow maintenance 
managers to design reports to provide them with the 
information they feel they need, and provide them with 
quicker feedback. A feasibility study examining the 
costs and potential benefits available from placing 
remote NALDA access terminals in every organizational 
level maintenance activity should be initiated. 

® A comprehensive examination of the factors critical to 
the success of maintenance managers should be conducted. 
This examination should seek to identify the specific 


factors, data elements, and information required for the 
knowledge base of the proposed expert systen. 


® A prototype expert system should be constructed to 
demonstrate the feasibility and benefits available from 
the use of such a system by maintenance managers. The 
break-out of NALDA data on a specific aircraft type, and 
the consolidation of this data into a separate database 
to be used for the ESAAMS prototype, is required. 
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